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EXECUTIVE  SUMMARY 

tJ 

a  Part  of  our  College  mission  is  distribution  of  the 
students’  problem  solving  products  to  DoD 
sponsors  and  other  interested  agencies  to 
^  enhance  insight  into  contemporary,  defense 
related  issues.  While  the  College  has  accepted  this 
product  as  meeting  academic  requirements  for 
'v?  graduation,  the  views  and  opinions  expressed  or 
implied  are  solely  those  of  the  author  and  should 
not  be  construed  as  carrying  official  sanction. 

~~“i insights  into  tomorrow” 

REPORT  NUMBER  88  -  0430 

AUTHOR(S)  MAJOR  RICHARD  S.  BUTLER,  USAF 
TITLE  air  force  systems  command  software  independent 

VERIFICATION  AND  VALIDATION  IMPLEMENTATION  ANALYSIS 
AND  GUIDANCE 


I.  Purpose ;  To  review  various  implementations  of  software 
Independent  Verification  and  Validation  (IV  St  V)  across  Air 
Force  Systems  Command  (AFSC) .  The  review  will  evaluate  the 
implementations  against  established  direction  and  propose,  if 
required,  alternatives  or  improvements  to  the  direction. 

II.  Probl em :  Software  IV  &  V  across  AFSC  is  implemented  based 
on  past  experiences  and  appears  to  deviate  from  established 
direction.  The  review  needs  to  assess  these  departures  and 
determine  what  common  concerns,  guidance  or  enhanced 
directions  are  required. 

III.  Anal ysi s;  There  are  six  product  divisions  within  AFSC 
that  implement  or  support  software  IV  St  V.  In  addition,  Air 
Force  Logistics  Command  supports  software  IV  St  V  requirements 
on  fielded  systems.  The  analysis  reviews  the  respective  IV  & 

V  implementations  and  support  actions  against  established 
direction.  Also,  the  particular  rational  for  each  application 
is  reviewed.  As  a  result  10  issues  are  identified  that 
concern  effective  IV  &  V  implementation.  These  10  issues  are 
grouped  into  two  Categories.  Category  I  concerns  lack  of 
common  software  IV  &  V  guidance.  Category  II  discusses  the 


need  to  obtain  IV  4<  V  data  in  developing  policy  which  is 
controllable  from  a  central  office. 

IV.  F i nd i nos i  Each  AFSC  and  support  area  follows  the  intent 
of  the  IV  6c  V  direction.  However,  a  need  exists  for  common 
guidance  to  bridge  the  gap  between  direction  and  application. 
This  provides  an  ability  to  concurrently  initiate  an  IV  6c  V 
effort  with  the  software  development  effort.  By  applying  IV  & 
V  guidance  early,  software  projects  may  avoid  problems  later 
on.  To  keep  guidance  effective  for  AFSC  a  central  office  is 
needed.  The  office  can  also  direct  that  IV  fit  V  be  considered 
for  projects  early.  This  consideration  happens  when  there  is 
direction,  time,  and  money.  A  central  office  using  feedback 
from  associated  AFSC  areas  establishes  interaction  among  the 
product  divisions  and  ability  to  develop  IV  fit  V  guidance  that 
is  supportive  of  the  field. 

V.  Cone  1  us i on  :  A  common  set  of  gu i de 1  ines  for  Category  I 
issues  are  developed.  The  guidance  covers  three  elements; 
general,  development  contractor,  and  IV  fit  V  agent.  This  way 
the  Government,  prime  contractor,  and  software  IV  fit  V  agent 
are  all  involved.  The  IV  fit  V  guidance  is  a  layered  approach, 
with  each  element  addressing  issues  for  effective  IV  fit  V 
interaction  and  evaluation.  To  support  Category  II  issues, 
four  tools  are  outlined  that  link  guidance/policy  to  field 
inputs  and  feedback.  The  tools  provide  the  data  base,  field 
concerns,  and  outside  evaluat  ons  that  allow  policy  to  reflect 
appropriate  needs.  Drawn  together,  the  guidance  and  tools 
provide  a  cohesive  software  package  that  supports  IV  fit  V 
implementations  across  AFSC. 

VI.  Recommendat ions:  Through  application  of  the  guidance  and 
tools,  AFSC  starts  towards  a  standard  software  IV  fit  V 
implementation.  The  process  starts  with  initial  guidance, 
allowing  the  AFSC  areas  to  improve  the  guidance  with 
headquarters  interaction,  not  interference.  The  improvements 
are  developed  using  existing  avenues  developed  for  each  tool. 
This  closes  the  loop,  allowing  software  IV  S<  V  to  be 
implemented,  evaluated,  enhanced,  and  re i mp 1 emen ted . 
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Chaptsr  One 

SOFTWARE  INDEPENDENT  VERIFICATION  AND  VALIDATION  (IV  &  V> 

BACKGROUND 

INTRODUCTION 


Software  Independent  Verification  and  Validation  is  a 
practice  which  presents  an  evaluation  of  software  products 
apart  from  the  original  developer's  evaluation.  The  practice 
involves  a  variety  of  techniques  which  evaluate  the  software. 
This  paper  will  not  debate  the  techniques  employed.  The 
purpose  is  to  review  software  IV  &  V  policies,  activities,  and 
implementations  across  Air  Force  Systems  Command  (AFSC). 
These  policies  will  then  be  assessed  against  AFSC 
implementations  for  inconsistencies  and  concerns  that  may 
require  new  or  updated  policies.  If  any  concerns  are  noted, 
they  will  be  evaluated  for  common  app 1 i cab i 1 i ty  across  AFSC ; 
along  with  specific  recommendations  for  software  IV  &  V,  in 
terms  of  policy  formulation  and  common  implementations,  if 
app 1 i cabl e . 

The  intended  audience  for  this  paper  is  the  Air  Force 
Systems  Command,  Mission  Critical  Computer  Resources  (MCCR) 
activities.  These  activities  are  located  throughout  AFSC  at 
the  product  division,  headquarters,  and  inspector  general 
field  offices.  Their  responsibilities  are  to  apply,  direct 
and  evaluate  MCCR  requirements  on  programs.  The  MCCR  primary 
objective  is  to  satisfy  the  system  level  sr  software  specific 
requirements.  One  avenue  supporting  this  objective  is  the 
application  of  software  IV  &  V  to  AFSC  programs. 


V* 


PROBLEM 


Implementation  of  software  IV  &  V  among  AFSC  product 
divisions  varies.  (27: — ;28: — ;29: — ; 30 : — ; 3 1 : — )  This 
variance  requires  review  to  ensure  that  implementations  comply 
with  prescribed  regulations  and  support  field  requ i remen ts . 
Additionally,  common  areas  of  concern  across  the  various 
implementations  require  identification.  These  common  areas 
require  evaluation  to  determine  if  a  common  set  of  software  IV 
&  V  guidance  or  requirements  can  be  developed  and  applied 
across  AFSC  implementations. 
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PREVIOUS  STUDIES 


Two  studies  performed  in  1981  and  1983  addressed  how 
software  IV  it  V  affected  programs.  Each  study  presented 
recommendations  to  improve  so-ftware  products  through  the  use 
of  IV  St  V.  These  studies  became  the  foundation  for  developing 
software  IV  &  V  direction. 

In  1981  Rome  Air  Development  Center  <RADC>  examined  the 
effects  of  IV  it  V  on  sof  tware  reliability,  ma  intainabi  1  i  ty, 
development  cost,  and  development  productivity.  Major 
conclusions  and  recommendations  are: 

1.  IV  &  V  significantly  improves  sof  tware  reliability. 
(4:39) 

2.  Cost  benefits  of  IV  &  V  are  enhanced  by  early 
detection  of  problems.  (4:89,100) 

3.  IV  &  V  should  be  used  early  to  detect  problems  and 
ensure  that  corrections  are  reverified.  (4:4,39) 

4.  Begin  IV  &  V  early  in  the  development  process  and 
require  delivery  of  preliminary  development  materials.  (4:80) 

The  Joint  Logistics  Commanders  Joint  Policy  Coordinating 
Group  on  Computer  Resource  Management  ( JLC-JPCG-CRM)  hosted  an 
Orlando  I  workshop  in  November  1983,  entitled  Post  Deployment 
Software  Support  (PDSS)  for  Mission-Critical  Computer 
Software.  IV  &  V  was  addressed  by  the  workshop  and  the 
findings  included: 

1.  "IV  &  V  can  and  should  be  used  in  all  phases  of  the 
software  development  cycle."  (1:2-5) 

2.  "IV  &  V  can  be  performed  "in  house"  or  with  a  separate 
contractor  as  long  as  the  IV  it  V  agent  is  independent  of  the 
developer."  (1:2-6) 

3.  Develop  a  policy  that  directs  the  program  managers  to 
determine  the  extent  of  IV  &  V  effort  to  be  used  on  their 
programs.  (1:2-7) 

4.  A  program  manager's  guidebook  is  needed  to  determine 
the  level  and  cost/benefit  of  IV  &  V  for  a  program.  Also, 
what  IV  &  V  specifics  should  be  accomplished  during  various 
phases  of  the  life  cycle.  (1:2-7) 


INTERIM  CONCLUSION 


The  key  conclusion  is  that  IV  &  V  does  improve  the  -final 
software  product.  The  improvement  is  beneficial  when  IV  &  V 
is  initiated  early,  before  the  design  is  firmly  established. 
<4:39,80)  Therefore,  a  policy  is  required  to  consider  IV  &  V 
implementation  early  in  a  project  to  support  early  detection 
of  errors.  In  addition,  software  guidance  is  needed  to 
implement  IV  &  V  effectively.  < 4 : 1 00 ; 1 : 2-7)  These  two 
recommendations  are  initiated  in  the  current  IV  &  V  related 
regulation  and  standards  reviewed  in  Chapter  Two. 


Chapter  Two 


CURRENT  DIRECTION  FOR  SOFTWARE  IV  &  V 

The  recommendations  in  Chapter  One  provided  the  baseline 
for  current  IV  &  V  direction.  This  direction  is  contained  in 
three  documents  which  address  application  and  implementation 
of  IV  &  V  for  software.  The  documents  are  AFR  800-14,  DOD- 
STD-21 67,  and  DOD-STD-2168  (Draft).  In  addition,  the  JLC- 
JPCG-CRM  is  developing  a  guidebook  for  software  IV  &  V 
applications  for  program  managers.  These  documents  are 
discussed  below. 


AFR  800-14 

AFR  800-14,  Lifecycle  Management  of  Computer  Resources  in 
Systems .  is  the  dominant  regulation  for  software  development 
within  Air  Force  Systems  Command.  The  regulation  recommends 
early  use  of  software  IV  &  V  to  detect  problems.  The  key 
points  are:  traceability  of  requ i remen ts ,  evaluations,  and 
testing.  These  points  are  noted  in  IV  &  V  definitions  and 
specifics  and  form  the  common  base  of  understanding  for  the 
application  of  IV  &  V  in  the  software  standards. 

Def i n i t i ons 


"Verification  is  an  evaluation  at  the  Computer  Software 
Configuration  Item  (CSCI)  level  to  determine  whether  the 
products  of  each  step  in  the  software  development  cycle 
fulfill  all  requirements  from  the  previous  step."  (2:21) 
Verification  is  initiated  as  early  as  possible,  normally 
during  the  requirements  analysis  stage,  to  ensure  requirements 
are  properly  documented  and  understood.  The  process  involves 
code  analysis,  evaluations  of  documents,  traceability 
matrices,  and  software  component  level  testing  to  ensure  lower 
level  requirements  are  properly  designed.  (2:21) 

"Validation  is  the  evaluation  and  testing  activities  at 
system  level  to  determine  compliance  of  the  final  CSCI  product 
with  the  system  requirements."  (2:21)  The  activity  covers 
actual  testing  of  code  against  a  set  of  requirements  or 
situations  involving  the  system.  Documentation  evaluation  and 
traceability  are  continued  against  the  evolving  code  and 
system  level  integration  requirements. 
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The  program  manager  will  consider  using  IV  &  V  based  on 
recommendations  from  the  Computer  Resources  Working  Group 
<CRWG>.  The  recommendations  are  documented  in  the  Computer 
Resources  Life  Cycle  Management  Plan  (CRLCMP).  (2:7)  The 
CRLCMP,  as  outlined  in  APR  800-14  requires  the  level,  scope, 
and  source  of  IV  &  V  described  within  the  quality  section, 
paragraph  8g,  o-f  the  CRLCMP.  (2:44)  In  addition,  four  key 
areas  supporting  the  implementation  of  IV  &  V  for  software 
require  addressing. 

1.  First  preference  for  an  IV  &  V  agent  should  go  to  the 
organization  supporting  the  software,  provided  the  required 
skills  and  resources  are  available.  In  any  case,  the  IV  &  V 
agency  should  be  separate  from  the  developing  agency,  to  avoid 
conflicts  in  the  independent  evaluation.  (2:7) 

2.  The  statement  of  work  (SOW)  for  the  developer  will 
require  granting  the  IV  &  V  contractor  access  to  the  software 
development  products  (code  and  documents).  In  addition,  any 
applicable  engineering  environments  for  the  various  levels  of 
testing  and  integration  tasks  will  be  made  available.  (2:10) 
Without  key  documents  and  software  environments  the  IV  &  V 
agent  will  not  be  able  to  evaluate  and/or  test  designs 
adequate  1 y . 

3.  Air  Force  Operational  Test  and  Evaluation  Center 
(AFOTEC)  or  designated  test  organization  will  “determine  the 
scope  and  nature  of  software  tests  for  the  Operational  Test 
and  Evaluation  (OT&E)."  (2:3)  The  test  organization  will 
provide  respective  inputs  to  the  Computer  Resources  Working 
Group  ( CRWG)  on  the  use  of  IV  &  V.  (2:3)  Without  test  inputs 
based  on  system  requ i remen ts  the  IV  &  V  agen t ' s  ab i 1  i t y  to 
support  testing  is  reduced.  The  IV  &  V  agent's  review  of  data 
helps  ensure  the  OT&E  can  function  as  expected. 

4.  The  Air  Force  Systems  Command/Air  Force  Logistics 
Command  supplement  to  AFR  800-14  adds  further  clarification 
for  IV  &  V  planning.  The  supplement  states  that  IV  &  V 
planning  will  be  completed  before  the  Full  Scale  Development 
(FSD)  Request  For  Proposal  is  released.  (3:11) 

AFR  800-14  establishes  the  goals  of  an  IV  &  V  effort  for 
software.  The  goals  are  implemented  through  tasking 
documents.  These  documents  are  the  standards  and  contract 
statements  employed  on  software  projects.  In  particular,  the 
software  development  standard,  DOD-STD-2167  and  associated 
data  requirements,  are  key  to  an  IV  &  V  effort.  These 
documents  outline  the  requirements  traceability,  evaluations, 
and  testing  for  software  products. 
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DOD-STD-21 67 ,  Defense  System  Software  Development.  4  June 
1985,  covers  the  entire  software  development  process  from 
concept  to  production.  The  standard  directs  the  developer  to 
coordinate  with  associated  contractors,  e.g.,  software  IV  6c  V 
agent.  (5:20)  DOD-STD-21 67A  (Draft,  1  April  1987)  addresses 
the  interface  with  the  software  IV  it  V  contractor  and  how  the 
developer  will  support  the  interface.  (21:11)  This  tasking 
between  the  developer  and  IV  &  V  agent  is  important.  It 
initiates  the  framework  under  which  the  products  and  processes 
are  obtained  and  evaluated.  Without  early  establishment  of 
responsibilities,  a  change  later  in  the  contract  may  cause 
cost  and  schedule  overruns. 

DOD-STD-2168  (Draft,  1  April  1987),  Defense  System 
Software  Quality  Program,  parallels  the  development  standard 
on  application  of  the  quality  evaluations  during  each  phase  of 
the  development  process.  In  particular,  DOD-STD-2168  supports 
DOD-STD-21 67A  (Draft)  on  interfacing  with  the  IV  &  V  agent  and 
requires  evaluations  of  the  corrective  action  system  under  the 
software  quality  program.  (23:4)  The  corrective  action  system 
is  where  IV  &  V  inputs  are  tracked  by  the  developer  and 
Government.  This  system  supports  RADC's  recommendation  of 
verifying  corrections  have  been  made. 


DI -MCCR-80030 ,  Software  Development  Plan  (SDP),  Data  I  tern 
Description  tasked  from  DOD-STD-21 67 ,  is  a  key  player  in  the 
software  development  process.  The  SDP  outlines  the  overall 
software  design  objectives  and  means.  It  requires  the 
developer  to  document,  in  part,  the  interface  with  the  IV  &  V 
agent  and  discuss  the  corrective  action  system.  (18:5,6)  By 
including  IV  &  V  data,  the  SDP  becomes  a  prime  management 
document  for  the  developer,  Government,  and  IV  &  V  agent. 

Through  contract  statements,  interactions  between 
standards  and  data  items  are  specified.  Without  the 
statements,  the  ability  to  tailor  specifics  of  the  standards 
and  related  data  items  for  various  applications  is  lost.  They 
also  outline  what  additional  requirements  are  needed  to  ensure 
the  Government  acquires  the  correct  products.  This  is  the 
link  that  ties  standards  and  data  items  to  the  IV  &  V  effort. 
Without  contract  statements,  the  prime  developer  will  not  know 
the  interactions  required  to  support  IV  &  V  tasks. 


IV  &  V  GUIDEBOOK 


— - - -  i 
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The  IV  6c  V  guidebook  is  currently  under  development  by  the 
JLC-JPCG-CRM .  The  guidebook  is  to  become  a  tool  for  program 
mangers  to  use  in  determining  the  extent  of  IV  &  V  application 
for  projects.  Future  sections  may  outline  SOW  descriptions 
that  could  aid  a  program  manager  in  structuring  a  contract. 

(17:1-1)  As  noted  above,  contract  statements  are  avenues  for 
proper  IV  &  V  tasking.  Without  guidance  to  program  managers, 
that  tasking  may  not  exist. 


INTERIM  CONCLUSION 


The  initial  studies  reflected  a  need  to  exert  software  3V 
&  V  early  in  software  development  projects.  The  purpose  is  to 
detect  problems  early  before  they  become  designed  into  the 
software.  The  regulation  and  standards  established  IV  &  V  as 
a  process  supporting  early  evaluation  of  software  designs. 

The  correlation  of  the  design  and  IV  &  V  process  are  derived 
from  the  contractual  statements.  The  key  task  is  to  derive 
specific  applications  of  software  development  and  quality  for 
the  particular  IV  &  V  implementation.  The  guidebook  is  one 
avenue  being  explored  now  by  the  JLC-JPCG-CRM.  However,  the 
guidebook  is  in  draft  and  currently  does  not  contain  guidance 
for  software  IV  &  V  tasking  within  statements  of  work  (SOW). 
The  intent  of  the  IV  &  V  direction  and  applicable  standards  is 
to  establish  an  early  link  between  development  and  evaluation 
of  the  software.  The  next  chapter  presents  how  this  link  is 
implemented  by  various  AFSC  product  divisions. 


Chapter  Three 


CURRENT  IMPLEMENTATIONS 

The  previous  chapters  outlined  the  problem  and  directions 
currently  established  -for  software  IV  &  V.  The  regulation, 
standards,  and  draft  guidebook  provide  initial  references  for 
the  program  manager.  However,  program  managers  lack  clear 
guidance  on  SOL)  tasking  for  IV  &  V  implementations.  To  better 
evaluate  this  lack  of  guidance,  a  review  of  AFSC  product 
division's  software  IV  &  V  implementation  is  needed.  This 
chapter  will  establish  whether  the  various  IV  &  V  approaches 
are  following  the  intent  of  existing  direction  specified  in 
Chapter  Two.  A  secondary  outcome  is  the  determination  if 
additional  policy  or  set  of  common  IV  &  V  requirements  are 
warr an  ted . 


INTERVIEWS 


The  following  data  was  collected  from  the  Mission-Critical 
Computer  Resource  (MCCR)  Focal  Points  in  the  respective  AFSC 
product  divisions.  Air  Force  Logistics  Command  was  also 
contacted  since  they  support  numerous  software  systems  and  act 
as  an  IV  &  V  agent.  The  respondents  indicated  their 
respective  areas  are  using  IV  &  V  on  major  programs.  The 
extent  of  and  arrangements  for  IV  &  V  are  based  on  system 
program  office  <SPO)  decisions  (acceptable  under  AFR  800-14, 
page  13).  The  decisions  are  the  result  of  recommendations 
provided  by  the  respective  MCCR  offices.  The  divisions  and 
AFLC  are  satisfied  with  their  respective  IV  &  V  efforts. 
However,  respondents  indicated  concerns  with  certain  aspects 
of  their  IV  &  V  implementations.  Each  IV  &  V  implementation 
and  associated  concerns  are  provided  below.  As  appropriate, 
the  respondent's  and  author's  concerns  are  blocked  and 
identified  with  issue  numbers  [ISSUE  #3.  This  allows  for 
grouping  the  issues  and  evaluation  in  the  interim  summary  and 
follow-on  chapters. 

Aeronautical  Systems  Division  <ASD) 

The  contracts  ASD  manages  normally  employ  the  prime 
developer  of  the  software  product  to  perform  IV  &  V  related 
tasks  internally.  This  is  allowed  under  AFR  800-14  page  21, 
provided  that  a  separate  organization,  apart  from  the  prime's 


development  team,  performs  the  required  tasks.  These  tasks 
and  organizational  setup  are  documented  in  the  Software 
Development  Plan  delivered  under  the  Request  for  Proposals 
(RFP)  and  contract.  <27:--)  This  approach  directs  the  prime 
to  document,  plan,  manage,  and  execute  a  verification  and 
validation  <V  &  V)  process  to  ensure  stated  requirements  are 
met.  With  the  developer  documenting  the  process,  the 
Government  can  evaluate  the  proposed  effort  during  contract 
award  and  track  the  actual  process  after  award.  This  internal 

V  &  V  process  is  one  element  in  the  ASD  software  integrity 
program  that  seeks  to  improve  operational,  supportable,  and 
reliable  software  in  weapon  systems.  <24: — ) 

C ISSUE  1]  Key  among  the  challenges  for  ASD  is  the 
selection  of  capable  software  developers,  application  of  a 
systematic  software  engineering  process,  and  development  of 
software  within  planned  program  baselines.  <27: — ) 

ASD  developed  the  integrity  program  to  address  issues 
affecting  weapon  system  software.  Various  systems  were  being 
delivered  with  software  errors,  inadequate  testing,  incomplete 
software,  and  developed  under  lax  engineering  practices. 

<27: — )  ASD's  effort  seeks  to  correct  these  deficiencies  by 
getting  developers  to  manage  and  implement  improved  software 
development,  support,  and  testing  processes.  The  internal  V  6c 

V  ties  the  processes  together  from  the  initial  software 
requirements  phase  to  final  customer  acceptance. 

ASD  pursued  an  internal  V  &  V  approach  since  previous  IV  6c 

V  results  were  unsatisfactory.  Previously,  documents  were  not 
totally  evaluated,  IV  &  V  experience  and  testing  was  lacking, 
and  development  contractors  were  relying  on  IV  6c  V  agents  to 
perform  the  developer's  software  engineering  and  quality 
responsibilities.  Since  ultimately  the  developer  is 
responsible  for  the  software  product,  not  the  IV  &  V  agent, 
the  onus  to  deliver  quality  software  and  ensure  it  meets  the 
requirements  belongs  to  the  developer. 

Internal  V  &  V  is  recommended  by  an  ASD  management  team  to 
help  new  programs  utilize  the  V  &  V  process  effectively  <this 
action  follows  the  guidance  in  AFR  800-14,  page  7).  The  team 
is  composed  of  experienced  software  acquisition  members  who 
evaluate  the  critical  nature  of  a  program's  software,  then 
propose  V  &  V  applications  and  tailoring  guidelines  to  the 
program  office.  Independent  V  6c  V  may  be  recommended  based  on 
the  critical  level  of  software;  i.e.,  safety  of  flight  or 
nuclear  essential  areas.  The  team's  recommendations  are 
provided  to  program  managers  for  inclusion  within  their  CRLCMP 
and  SOW.  In  addition,  the  team  recommends  software 
development  specifics  for  the  RFPs,  completing  the  software 
integrity  structure  for  each  program.  <27: — ) 
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The  ASD  approach  allows  the  developer  and  outside  V  Sc  V 
agent  to  agree  on  the  availability  and  proprietary  nature  of 
any  of  the  development  that  must  be  used  to  properly  test  and 
evaluate  the  software.  This  last  area  is  critical  when 
considering  the  competitive  edge  a  software  developer  wants  to 
maintain.  Therein  lies  the  problem  of  competing  software 
developers  performing  IV  Sc  V  on  the  other's  product.  The 
final  effect  may  be  higher  costs  to  offset  a  potential  loss  of 
market  share  to  the  developer. 

[ISSUE  21  Without  a  contractual  understanding  between  the 
developer  and  V  Sc  V  agent,  the  agent  may  gain  an  unfavorable 
advantage  over  the  developer  on  future  contracts.  [Author's 
Issue) 


[ISSUE  31  There  is  the  possibility  the  developer  will 
not  allow  the  IV  Sc  V  agent  proper  access  to  key  software 
elements,  opening  the  potential  for  untested  software  in  the 
weapon  system.  (Author's  Issue) 

Under  AFR  800-14,  page  10,  the  agent  must  have  required 
access  to  the  needed  elements.  The  ASD  approach  allows  the 
developer  to  acquire  V  &  V  resources  internally,  while  still 
holding  the  developer  responsible  for  overall  quality  and 
performance  of  the  software.  The  ASD  software  V  &  V 
implementation  follows  the  intent  of  the  IV  &  V  direction. 


Electronic  Systems  Division  (ESQ) 


ESD  employs  an  8a  firm  (small  disadvantaged  business)  to 
provide  an  “on-call"  (available  as  needed)  IV  &  V  capability 
for  ESD  program  offices.  The  program  offices  determine  the 
applicability  of  IV  Sc  V  based  on  recommendations  from  the  CRWG 
and  associated  offices  within  ESD.  The  program  offices  using 
the  "on-call"  IV  &  V  agent  fund  their  respective  portion. 

(30  :  — ) 


[ISSUE  4]  The  ability  to  determine  initial  tasking  and 
cost  estimating  between  program  office  and  agent's  contract 
requirements  is  needed.  (Author's  Issue) 

The  "on-call"  approach  was  developed  due  to  IV  &  V  being 
initiated  late  in  many  ESD  programs,  usually  after  errors  had 
been  detected.  The  IV  Sc  V  agent  performs  documentation  and 
testing  as  required  by  program  offices.  Skill  and  experience 
level  of  the  agent  is  being  improved  by  employing  experienced 
IV  Sc  V  individuals.  ESD  meets  the  intent  of  the  IV  Sc  V 
direction  with  this  approach. 


[ISSUE  51  The  13  individuals  may  be  tasked  to  support 
multiple  ESD  programs,  limiting  their  availability  to  support 
any  one  program  full  time.  (Author's  Issue) 

[ISSUE  61  Since  this  is  a  new  approach,  a  survey  should 
be  initiated  to  monitor  IV  &  V  implementation,  impacts,  and 
improvements  in  order  to  determine  the  utility  of  the  IV  &  V 
program.  <30: — ) 

Space  Division  <SD) 

SD  is  using  IV  &  V  on  major  programs  such  as  NAVSTAR, 

Boost  Surveillance  Tracking  System  (BSTS),  and  Milstar.  Each 
program  is  using  separate  IV  &  V  contracts.  Each  program  uses 
recommendations  from  their  software  project  manager  and 
computer  resource  office  to  implement  IV  &  V.  The  rational 
for  separate  contracts  is  that  complexity,  size,  and 
uniqueness  of  each  program  requires  an  IV  &  V  agent  dedicated 
to  the  project.  <31: — )  SD's  implementation  follows  the 
intent  of  the  IV  &  V  direction. 

SD  indicated,  from  their  experience,  some  development 
contractors  rely  on  IV  &  V  agents  to  catch  problems  the 
developers  should  have  earlier.  This  allows  the  developer  to 
meet  schedules  and  worry  about  product  quality  later. 

[ISSUE  71  To  counter  the  above  attitude,  improved 
software  management  techniques  are  required  in  the  developer's 
and  IV  &  V  agent's  SOW.  <31:  — ) 

Air  Force  Contract  Management  Division  <AFCMD) 

AFCMD  noted  that  numerous  programs  they  monitor  employ  IV 
&  V  tasks.  They  discovered  certain  tasks  requested  under  the 
auspices  of  IV  &  V  were  really  Contract  Administration  Service 
<CAS>  functions.  They  resolved  the  duplications  and  improved 
the  IV  6i  V/CAS/SPO  relationships.  <32: — )  AFCMD's  interaction 
with  IV  &  V  follows  the  applicable  direction.  They  did  raise 
two  issues  as  a  result  of  the  duplication. 

[ISSUE  8]  AFCMD's  primary  concern  is  the  contractual 
implementation  of  the  IV  &  V  task  must  avoid  duplication  of 
effort  between  Government  and  IV  &  V  agent.  <32: — ) 


[ISSUE  6.1]  A  survey  of  AFSC  IV  6c  V  efforts  could  help 
avoid  duplications.  The  survey  should  identify  IV  &  V  areas 
used  in  supporting  weapon  systems  and  interactions  between  the 
IV  &  V  agent,  CAS,  and  SPO  during  software  development.  The 
results  could  provide  insight  on  how  to  structure  contracts  to 
gain  the  best  IV  &  V  effort  given  various  circumstances. 

<32: — )  This  issue  is  common  to  the  ESD  Issue  6. 
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Ballistic  Missile  Organization  (BMP) 

BMO  uses  IV  4c  V  on  an  extensive  basis  under  competitive 
procurements.  Their  agents  have  a  working  knowledge  of  past 
BMO  requirements  and  provide  both  documentation  and  testing 
capabilities.  IV  4c  V  is  mandated  on  programs.  A  majority  o-f 
these  programs  require  nuclear  certification;  thus,  software 
is  evaluated  to  ensure  requirements  are  correct  and  met.  No 
concerns  were  reported.  (29: — )  BMO  -follows  the  intent  o-f  the 
IV  &  V  di rect i on . 

[ISSUE  9]  A  minimum  set  o-f  IV  4c  V  criteria  -for  updating 
established  parameters  and  projects  is  essential  even  under 
strict  requirements.  (Author's  Issue) 

Armament  Division  (AD) 

AD  is  implementing  IV  &  V  on  major  programs.  The 
implementation  is  based  on  CRWG  and  computer  resource  office 
recommendations  to  the  program  offices.  The  recommendations 
are  forwarded  to  the  SPO  for  consideration  during  the 
development  of  the  FSD  Request  For  Proposals.  The  Advanced 
Medium  Range  Air  to  Air  Missile  (AMRAAM)  is  using  the  Navy  at 
Point  Magu  and  Air  Force  at  Warner  Robins  Air  Logistics  Center 
as  IV  &  V  agents.  AD  has  implemented  IV  4c  V  on  the  Training 
Ranges  segment  of  the  Global  Positioning  System  program.  In 
both  instances  the  IV  &  V  effort  is  reported  as  satisfactory. 
(28s  — )  AD's  implementation  follows  the  intent  of  the  IV  4c  V 
direction. 

[ISSUE  103  However,  AD  focal  point  noted  that  IV  4c  V  is 
normally  advocated  by  them,  instead  of  collectively  with  the 
using  and  supporting  commands.  Therefore,  advocacy  for  IV  4c  V 
from  higher  echelons  could  improve  IV  4c  V  use.  (28:  — ) 

Air  Force  Looi sties  Command  (AFLC) 

AFLC  confirmed  that  they  are  providing  IV  4c  V  support  on 
numerous  projects,  e.g.  Bl-B  Automatic  Test  Equipment  and  F- 
16A/B  Operational  Flight  Program  (OFP).  The  F-16  IV  4c  V 
effort  on  OFPs  was  successful.  The  knowledge  gained  by  the  IV 
4c  V  agent  (Ogden  Air  Logistics  Center  <ALC))  and  developer 
(General  Dynamics)  allowed  each  to  switch  roles;  Ogden  ALC 
performs  F-14  A/B  OFP  block  updates  and  General  Dynamics  is 
the  IV  4c  V  agent.  (33: — >  AFLC  follows  the  direction  within 
AFR  800-14,  since  they  perform  IV  4c  V  and  maintenance  roles 
for  various  weapon  systems.  (2:7) 
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CISSUE  6.2]  AFLC  is  developing  a  survey  to  collect  IV  Sc  V 
data  concerning  AFLC  products.  (33 — )  The  data  could  support 
similar  efforts  at  AFSC  and  is  a  companion  to  ESD  Issue  6  and 
AFCMD  Issue  6.1.  (Author's  Issue) 

Summar  y 

The  review  identified  ten  issues  that  effect  IV  t*  V 
implementations  across  AFSC.  These  issues  are  grouped  under 
two  categories  for  software  IV  &  V.  as  summarized  in  Table  1. 
Category  I  involves  the  availability  of  specific  guidance  to 
the  Government,  developer,  and  agent  in  terms  of  data, 
information,  and  levels  of  testing  that  are  required  for  IV  & 
V.  Category  II  involves  initiating  IV  &  V  early  in  the 
program's  life  cycle.  Without  early  IV  &  V  the  ability  to 
detect  errors  is  lost.  Early  implementation  requires  an 
advocate  for  IV  &  V.  The  advocate  ensures  IV  &  V  is 
considered  at  the  decision  making  levels  on  par  with 
development  decisions.  The  computer  resource  focal  points 
stated  that  AFSC  is  usually  the  early  advocate  for  IV  &  V. 
Therefore,  when  IV  &  V  is  used,  it  is  due  in  part  to  the 
ability  of  the  AFSC  computer  resource  offices  and  engineers 
who  recommended  implementation  of  IV  &  V  to  the  program 
manager.  Additionally,  a  strong  advocate,  monitoring  the  IV  & 
V  tasking,  provides  impetus  to  use  and  support  the  additional 
cost  and  schedule  drivers  required  to  perform  IV  &  V. 


INTERIM  CONCLUSION 

Based  on  the  above  implementations  and  governing 
directions,  the  AFSC  product  divisions  are  implementing  IV  &  V 
correctly.  They  recognize  IV  &  V  is  advantageous  and  provides 
the  program  offices  with  abilities  to  improve  reliability 
through  detecting  problems  earlier  in  the  software 
requirements  and  code  development  cycles.  Each  implementation 
is  within  the  scope  of  the  rules  established  under  AFR  800-14. 
As  more  product  divisions  begin  implementing  DOD-STD-2167  on 
new  programs,  the  respective  IV  &  V  tasking  will  improve,  but 
differently  from  other  divisions.  Additional  IV  &  V  policy, 
beyond  AFR  800-14's,  is  not  required.  What  is  required  is 
guidance  on  initial  common  approaches  and  a  central  office  for 
collecting  IV  &  V  implementation  data  and  advocating  its  use. 

To  provide  the  initial  approach  and  a  central  office,  the 
two  categories  above  offer  avenues  to  achieve  these  needs. 
Category  I  concerns  various  unstructured  implementations  of 
software  IV  &  V  across  AFSC.  Even  with  improved  IV  &  V 
tasking,  as  more  divisions  adopt  DOD-STD-2 1 o7 ,  there  is  no 
direct  IV  &  V  guidance.  A  common  set  of  software  IV  &  V 
guidance  provides  a  foundation  for  each  product  division  to 
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Category  I  -  Specific  Guidance  for  Software  IV  &  V 

Issue  # 

Division 

Issue  Explanation 

Selection  of  competent  software 
contractors,  following  established 
processes . 

Establish  understanding  between 
software  developer  and  IV  6c  V  agent 

Provided  required  access  to  materials 

Provide  "on-call"  IV  &  V  assistance. 

Support  multiple  programs  with  fixed 
resources . 

Avoid  using  IV  &  V  as  developer's 
quality  function.  Improved  software 
management  techniques  are  required. 

Implement  IV  &  V  correctly  on 
contract,  avoid  duplication  w/AFCMD 
f unc  t i ons . 

IV  4  V  used  extensively,  no  tailoring 
required. 


Category  II  -  Software  IV  &  V  Data  Collection/Advocacy 

Issue  # 

Division 

Issue  Explanation 

<6 

ESD 

Survey  to  monitor  effectiveness  of 
"on-call"  IV  &  V  agent  for  ESD 

6.1 

AFCMD 

Identify  interactions  with  parties 

6.2 

AFLC 

Separate  survey  for  IV  6c  V. 

7 

SD 

Improve  software  management. 

10 

AD 

Maintain  software  IV  6c  V  advocacy. 

Table  1.  Software  Categories/Product  Division  Issues 
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initiate  their  own  independent  software  evaluations.  They 
currently  lack  a  reference  that  lists  the  minimum  efforts  for 
software  IV  &  V.  The  guidebook  being  developed  by  the  JLC- 
JPCG-CRM  may  cover  these  efforts,  but  is  not  yet  published. 
(34: — )  Therefore,  the  need  for  guidance  is  urgent  as  more 
product  divisions  adopt  the  software  development  philosophy  of 
DOD-STD-21 67 . 

Category  II  covers  the  requirements  for  IV  &  V  data 
collection  and  tracking  to  ensure  proper  implementation.  The 
data  would  provide  an  assessment  of  IV  &  V  policy  and  tactics 
to  the  central  office.  This  office  serves  as  the  central 
focal  point  for  developing  a  coordinated  policy  and  as  the 
advocate  for  IV  Sc  V.  This  in  turn,  provides  product  divisions 
a  central  authority  on  policy  that  works  with  field  users  to 
promote  positive  effects  of  IV  Sc  V  policy  and  application. 

The  next  chapter  outlines  each  category's  issues  and 
solutions. 
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I. 

|  Chapter  Four 


COMMON  SOFTWARE  IV  6c  V  REQUIREMENTS 

Chapter  Three  reviewed  various  IV  4<  V  implementations 
across  AFSC  Product  Divisions.  The  review  concludes  that  each 
implementation  is  -following  the  intent  o-f  the  governing 
regulations.  However,  10  issues  surfaced  that  concern  IV  it  V 
implementations.  Those  issues  fall  within  one  of  two 
categories,  summarized  in  Table  1.  The  issues  do  not  violate 
the  governing  regulation,  so  the  need  for  additional 
regulations  is  not  warranted.  What  is  necessary  is  common 
guidance  and  support  for  software  IV  6c  V.  Therefore,  an 
evaluation  of  each  category,  associated  issues,  and 
recommended  solution  for  IV  &  V  guidance  and  support  is 
provided  below. 

Category  I  covers  those  issues  needing  guidance  on  the 
application  of  software  IV  6c  V  for  programs.  Category  II 
covers  those  issues  involving  the  collection  of  data  and 
support  for  IV  6c  V  across  AFSC.  Category  evaluations  are 
based  on  numerous  sources  and  the  authors  12  years  of  software 
related  acquisition  experience,  which  included  experience  in 
the  drafting  of  DOD-STDs-21 67 ,  -2167A  (Draft),  -2168  (Draft) 
and  associated  policies. 


CATEGORY  I 

SOFTWARE  IV  &  V  GUIDANCE 

Category  I  addresses  the  need  for  sufficient  guidance  to 
allow  product  divisions  to  initiate  IV  &  V  efforts  early  to 
support  a  program's  development  effort.  Issues  from  five 
product  divisions  addressed  this  need,  as  outlined  in  Table  1. 
These  issues  are  evaluated  below  along  with  a  proposed 
solution  for  Category  I. 

The  Category  I  dilemma  is;  to  initiate  IV  6c  V  efforts 
early  requires  the  product  divisions  to  know  IV  &  V 
requirements  for  the  software  development  project  being 
proposed.  This  forces  program  offices  to  understand  software 
and  hardware  interrelationships  within  the  development  effort 
before  they  are  developed.  Knowing  interrelationships 
beforehand  is  difficult,  resulting  in  unknown  requirements 
surfacing  and  effecting  contracts,  schedules,  and  performance 
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in  a  negative  manner.  However,  a  minimal  set  of  IV  &  V 
requirements  can  establish  the  ground  work  -for  both  the 
developer's  and  IV  tc  V  agent's  proposals  and  designs.  Without 
established  requirements  in  the  RFP,  IV  6t  V  tasks  lag  the 
development  effort  causing  probable  delays  in  the  end  product. 
The  developer  will  in  turn  be  told,  after  development  starts, 
what  is  expected  in  terms  of  IV  &  V  support,  leading  to 
additional  cost  and  schedule  factors  that  in  all  probability 
were  not  accounted  for  in  the  winning  proposal . 

Product  Division  Issues 

There  are  eight  issues  that  comprise  Category  I.  Each 
issue  has  concerns  requiring  satisfaction  before  the 
particular  division  can  better  implement  software  IV  &  V.  A 
common  theme,  among  the  issues,  is  the  availability  of 
guidance  to  improve  IV  t*  V  implementation.  The  discussion 
below  outlines  the  issue's  requirements  to  improve  the 
implementation  and  type  of  guidance  required. 

ASP  Issues  1.  2.  and  3.  Aeronautical  Systems  Division's 
issues  require  specific  guidance  in  order  to  ensure  their 
products  are  adequately  assessed.  The  guidance  should  cover 
either  an  internal  (developer)  or  external  IV  6c  V  agent.  An 
evaluation  of  these  issues  is  presented  below. 

I ssue  1 .  ASD  needs  to  ensure  software  developers  are 
capable,  are  using  systematic  software  engineering  processes 
< DOD-STD-21 47) ,  and  the  development  is  within  established 
baselines.  Without  initial  guidance  the  possibilities  of 
inconsistencies  between  the  development  contract  and  IV  6c  V 
effort  will  develop.  As  a  result,  the  probability  of 
evaluating  outdated  products,  documents  or  out  of  scope 
requirements  increases.  Guidance  on  IV  &  V  interactions 
between  the  developer  and  IV  Sc  V  agent  could  lower  the 
probability  of  inconsistencies  and  improve  the  engineering 
process . 

In  contracts,  the  developer  must  indicate  the 
relationships  with  any  outside  contractor  or  requ i remen ts .  If 
a  common  set  of  software  development  related  IV  &  V 
requirements  are  provided,  the  developer's  reply  will  indicate 
his  understanding  of  the  engineering  processes  and 
interactions  required.  An  IV  &  V  set  aimed  at  the  developer 
provides  initial  data  allowing  the  program  offices  to 
determine  if  developers  are  capable  of  performing  software 
development  and  IV  &  V  efforts  as  applicable. 

I ssue  2 .  In  conjunction  with  Issue  1,  the  second 
concern  is  the  understanding  between  the  developer  and  IV  &  V 
agent.  This  understanding  is  the  contractual  language  that 


directs  each  party  to  provide  specific  information  to  the 
other  without  threat  of  competition.  Issue  1  involves 
providing  the  developer  with  initial  IV  &  V  guidance  so  the 
program  office  could  adequately  evaluate  the  proposal.  In 
turn,  the  IV  &  V  agent(s)  require  a  common  set  of  tasks  for 
their  evaluation  and  response  to  parallel  the  developer's 
during  the  same  period.  Without  these  common  tasks,  the  agent 
is  still  evaluating  his  response  while  the  development  is  into 
critical  design.  Therefore,  to  ensure  the  IV  6c  V  agent  is  on¬ 
line  with  the  development  schedule,  a  common  set  of  IV  4c  V 
requirements,  established  early,  are  required.  Without 
coordinated  contractual  efforts,  between  developer  and  agent, 
assumptions  are  made  on  responsibilities  and  relationships.  A 
concurrent  set  of  IV  &  V  agent  tasks  specify  the  initial 
relationships  and  responsibilities.  This  allows  the 
Government,  developer,  and  agent  to  coordinate  during  contract 
formulation  on  what  is  expected.  This  helps  in  avoiding 
conflicts  of  interest.  The  added  benefit  provides  the  ability 
to  cross  check  requ i remen ts .  Thereby,  avoiding  tasking  the 
agent  to  check  a  function  that  is  not  required  or  proprietary. 

Issue  3.  The  final  issue  concerns  access  to  required 
materials.  The  developer  and  agent  must  have  access  to  the 
other's  materials,  as  contractually  specified.  This  must  be 
dealt  with  up  front  in  each  contract.  By  providing  guidance 
on  initial  IV  &  V  and  development  interactions,  each 
contractual  party  can  propose  guidelines  for  release  of 
materials,  schedules,  and  cost  for  the  access.  This  allows 
the  Government  the  ability  to  evaluate  the  responses  to  ensure 
that  required  access  is  available,  requested,  and  does  not 
duplicate  existing  resources  available  to  the  program  office. 

ESP  Issues  4  and  5.  Electronic  Systems  Division's  issues 
require  review  to  ensure  "on-call"  resources  are  efficiently 
employed.  General  guidance  on  how  program  offices  integrate 
the  resources  within  the  programs  is  required.  This  allows 
ESD  to  fully  utilize  the  "on-call"  capabilities  of  the  IV  &  V 
agent.  The  following  discussion  outlines  the  concerns. 

Issue  4.  ESD  uses  an  "on-call"  IV  &  V  agent  for  use 
across  ESD  programs.  Each  program  office  determines  the 
utility  of  using  the  agent,  within  the  confines  of  the  agents 
contract.  Since  each  office  funds  their  use  of  the  agent, 
they  require  an  initial  estimate  of  the  cost.  Common  guidance 
for  IV  &  V  tasking  could  provide  initial  estimates.  Common 
guidance  provides  the  ability  to  enhance  or  tailor  out 
specific  tasks.  This  approach  allows  improved  cost  estimates 
since  the  agent's  overall  contract  contains  related  costs  for 
tasks  and  man-hours.  (30:--) 


Issue  5.  An  “on-call"  IV  fit  V  agent  -for  ESD  presents 
a  serious  problem.  With  only  a  limited  number  of  personnel 
the  agent  may  not  spend  sufficient  time  on  all  IV  &  V  tasking. 
This  requires  the  program  offices  and  ESD  MCCR  focal  point 
to  ensure  tasks  are  adequately  covered.  If  not,  then 
additional  manpower,  separate  contracts,  or  elimination  of  IV 
6c  V  tasks  are  required.  Without  an  initial  set  of 
requirements,  the  agent  may  be  tasked  with  a  larger  role  than 
is  feasible.  The  result  is  software  not  properly  evaluated, 
evaluated  too  late,  or  not  at  all;  which  affects  cost, 
schedule,  and  performance.  A  common  set  of  requirements 
provide  the  ability  to  tailor  tasks  and  determine  availability 
and  credibility  of  resources  to  ensure  tasks  are  adequately 
covered . 

SD  Issue  7.  Various  IV  &  V  implementations  within  Space 
Division  have  become  crutches  for  the  software  developers. 

The  developers  use  the  IV  &  V  agent  as  the  prime's  quality 
evaluator,  relieving  the  developer  of  responsibility  to 
properly  check  the  product.  This  causes  agents  to  be  drawn 
off  from  true  IV  fit  V  tasking.  Therefore,  SD  requires  better 
software  management  practices  earlier  in  the  development  cycle 
to  avoid  this  problem.  The  role  of  the  IV  &  V  agent  is  to 
ensure  requirements  are  being  met  in  both  the  validation  role 
and  verification  methods.  A  common  set  of  requirements  for 
the  agent  would  allow  enhancement  or  tailoring  of  requirements 
to  track  a  specific  development  effort.  In  addition,  specific 
areas  are  required  of  the  developer  to  ensure  they  remain 
responsible  for  the  final  product  and  interact  properly  with 
the  agent.  SD  has  addressed  their  concern  by  developing  the 
Software  IV  fit  V  Guidebook.  The  guidebook  covers  areas  of 
assessment  for  various  types  and  critical  levels  of  software 
based  on  proposed  application.  <26: — )  At  this  writing,  it 
does  not  cover  i ncor por at i on  of  DOD-STD-21 67 .  A  generic  set 
of  IV  fit  V  requirements  in  conjunction  with  DOD-STD-2167  would 
aid  the  initial  software  management  improvement  issue  and  an 
updating  of  the  guidebook  later. 

AFCMD  Issue  8.  Numerous  software  evaluations  on  a 
contract  require  Government  action,  not  contractor  action. 

Air  Force  Contract  Management  Division's  concern  with  IV  &  V 
implementation  is  duplication  with  AFCMD's  responsibilities. 
AFCMD  supports  and  directs  Government  offices  in  reviewing 
contractor's  performance  and  aids  program  offices  in  managing 
the  effort.  The  AFCMD's  functions  often  involve  reviewing  key 
documents  and  recommending  specific  actions  to  program 
offices.  AFCMD  has  found  certain  of  these  functions  being 
performed  by  IV  fit  V  agents.  This  is  duplicative  and  costly, 
since  AFCMD  has  resources  to  perform  the  functions  at  no 
additional  cost  to  the  Government.  Additionally,  AFCMD  has 
resources  on-site  or  available  within  a  region  to  interact 
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with  the  developer.  This  removes  the  obstacle  of  conflict  of 
interest  and  availability  of  resources  between  developer  and 
Government.  To  ensure  duplication  is  avoided,  a  common  set  of 
IV  6c  V  requirements  coordinated  along  with  the  development 
contract  with  AFCMD  is  warranted.  The  coordination  of  effort 
is  essential  and  is  commonplace  among  the  chief  participants 
(program  office,  contracting  agencies  (AFCMD),  potential 
bidders  (development  and  IV  &  V) ,  etc.).  Therefore,  common, 
coordinated  IV  &  V  guidance  establishes  primary  offices'  of 
responsibility's  level,  depth,  and  interaction  on  a  contract. 

BMP  Issue  ?■  The  Ballistic  Missile  Office  IV  Sc  V 
implementation  presents  a  unique  concern  from  the  author's 
perspective.  Software  IV  &  V  is  used  extensively  to  evaluate 
BMO  projects.  However,  the  author's  concern  is  not  every 
project  requires  extensive  IV  &  V.  There  are  times  when  a 
minimum  IV  &  V  set  is  useful,  e.g.,  updating  or  minor 
corrections  to  established  programs.  Therefore,  a  common  set 
provides  general  tasking  which  may  be  tailored  to  f i t  known 
corrections  and  baselines. 

Category  I  Requirement 

Based  on  the  above  issues,  the  need  for  a  skeleton 
structure  to  develop  IV  &  V  guidance  and  requirements  is 
paramount.  The  structure  allows  the  IV  &  V  proposal  to 
develop  during  the  same  period  as  the  software  development 
proposal.  Drawing  on  the  initial  intent  of  the  Government's 
development  requirements,  the  IV  &  V  effort  will  parallel  top 
level  tasks.  The  key  is  flexibility  in  establishing  the 
initial  steps  and  allowing  future  growth  or  tailoring  of 
contract  requirements.  The  proposed  solution  for  a  flexible 
IV  &  V  structure  is  listed  below. 

Category  I  Proposed  Solution  -  Guidance  for  Software  IV  &  V 


To  obtain  a  flexible  IV  &  V  structure  requires  common 
guidance  for  IV  &  V  requirements  be  available.  To  develop 
common  guidance  three  assumptions  are  made.  First,  each 
product  division  will  implement  software  development  under 
DOD-STD-21 67 ,  as  mandated  by  HQ  AFSC,  for  all  new  programs 
effective  March  1986.  (25: — )  This  allows  IV  &  V  requirements 
to  be  traced  directly  to  the  software  development.  Two,  the 
developer  and  IV  6c  V  agent  will  interact,  by  contract 
declaration,  to  review  the  products  and  other  material  as 
specified  to  the  depth  specified.  Three,  in  the  absence  of 
any  direct  standard  or  regulation  on  software  IV  6c  V 
requirements,  guidance  is  needed  to  direct  IV  &  V 
implementation  in  accordance  with  DOD-STD-21 67 .  Therefore, 
the  proposed  solution  is  divided  between  general  and  specific 
IV  &  V  guidance.  The  general  guidance  covers  common  areas  the 
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Gove r nme n t  should  address.  The  specific  guidance  is  divided 
into  two  sections;  1>  Specific  Guidance  and  Tasks  for 
Developer  and  2)  Specific  Guidance  and  Tasks  for  IV  &  V  agent. 

General  Gu i dance .  The  Government  must  emphasize  in  the 
Request  For  Proposals  (RFP)  the  objective  in  establishing 
requirements,  documentation,  quality  tasks,  and  IV  4<V  on  the 
project.  In  the  author's  view  the  objective  is  to  produce  a 
common  thread  of  tracking  requirements  from  inception  to 
production,  fielding,  maintenance,  and  final  retirement.  If 
there  are  two  separate  parties,  developer  and  agent,  then  the 
RFPs  will  be  separate.  RFP  coordination  is  essential  at  this 
point.  The  coordination  ensures  requirements  are  accurate  and 
supported  by  each  RFP.  The  contracts  should  be  written  and 
reviewed  by  the  same  evaluation  members  of  the  Government  to 
ensure  requirements  are  not  conflicting,  redundant,  or  non¬ 
existing  in  one  contract  but  required  in  the  other.  Without 
effective  controls  the  Government  looses  oversight,  developer 
looses  the  objective,  and  the  IV  &  V  agent  will  flounder  using 
older  requirements  against  newer  data  that  may  not  correlate 
to  test  requirements  or  results. 

Tailoring  of  requirements  is  expected.  However,  if 
tailored  requirements  are  not  correlated  between  development 
and  IV  &  V  contracts,  the  effects  in  the  above  paragraph  will 
surface.  Key  drivers  in  tailoring  are  the  operational 
environment  and  projected  support  r equ i r emen ts .  Without 
initial  knowledge  of  these  areas  tailoring  tends  to  eliminate 
tasks  that  support  the  drivers.  If  left  unchecked  tailoring 
erodes  the  product  development  and  software  integrity.  To 
sacrifice  an  item  for  cost  and  schedule  improvements  could 
undermine  software  performance  to  the  point  of  unworkable, 
unsuppor tabl e  and  probably  unmanageable  system  level  concerns. 
(4:39)  The  end  result,  in  the  author's  experience,  is  higher 
development  costs  and  longer  schedules  to  correct  issues  that, 
if  tailored  properly  would  have  been  manageable,  not 
necessarily  unavoidable. 

Key  in  any  development,  especially  with  multiple 
contracts,  is  a  common  set  of  terms,  and  definitions  for  the 
project.  Along  with  common  terms  the  Government  must 
establish  and  understand  the  main  software  drivers.  Those 
drivers  consist  of  Computer  Software  Configuration  I  terns 
<CSCI)  and  the  related  interfaces  at  the  macro  level.  Without 
the  interfaces  and  CSCIs,  the  common  thread  for  the 
development  is  harder  to  establish.  The  thread  provides 
traceability  among  the  software  drivers  back  to  the  system  or 
sub-system  requirements.  In  addition,  top  level  software 
drivers  can  trace  requirements  down  to  actual  code  level 
implementations,  if  low  level  specifications  are  required. 
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It  is  critical  that  the  agent,  Government,  and  developer 
understand  the  thread  of  continuity  for  the  software.  A 
method  to  establish  continuity  is  effective  application  o+ 
DOD-STD-21 67 ,  MI L-STDs-490 ,  -483,  and  -1521.  The  application 
allows  tailoring  of  documentation,  specifications,  testing, 
and  reviews.  These  four  standards  contain  software 
development  and  IV  &  V  related  efforts.  The  successful 
integration  of  the  software  tasking  achieves  the  continuity 
and  positive  interactions  between  all  participants. 

Finally,  avoid  excessive  changes  in  the  developers 
contract;  which  differ  from  the  initial  requirements  in  the 
RFP.  Since  the  objective  is  parallel  contracts,  excessive 
changes  require  recoordi nat i on  of  tasks  between  developer  and 
IV  &  V  contracts.  One  approach  is  to  release  the  agents  RFP 
during  the  developer's  Best  And  Final  Offer  <BAFO>.  This 
allows  the  IV  6c  V  RFP  to  contain  a  majority  of  final  tasks 
that  support  the  development  effort. 

Specific  Guidance  and  Tasks  for  Developer.  The  specific 
guidance  a  developer  must  know  is  that  an  IV  &  V  agent  or 
activities  will  be  required.  This  is  critical  even  if  the 
developer  and  IV  &  V  agent  are  the  same.  Without  specific 
guidance,  the  internal  or  external  teams  will  not  understand 
responsibilities,  limitations,  and  requirements.  This  allows 
the  developer  to  structure  the  proposal  to  reflect  IV  &  V 
related  tasks.  Those  developer  related  tasks  are  outlined 
below. 

Developer  Specific  Tasks.  Specific  contractual  tasks 
are  required  to  ensure  the  developer  understands  the  basics  of 
software  development  and  evaluation.  The  following  represents 
a  common  set  of  IV  &  V  related  tasks  a  developer  should 
perform.  Without  a  common  set,  the  interaction  between  the 
developer,  Government,  and  IV  &  V  agent  is  jeopardized. 

1.  The  developer  is  required  to  develop  and 
implement  a  Software  Quality  Program  and  document  the  program 
in  a  Software  Quality  Program  Plan  <SQPP>  or  Evaluation  Plan 
<SQEP).  The  plan  contains  various  IV  &  V  activities.  <19:6; 
7:9)  This  brings  division  of  responsibilities  between  the 
developer  and  agent  up  front.  Additionally,  the  developer  is 
tasked  with  developing  the  Software  Development  Plan  (SDP) , 
which  documents  relationships  and  interactions  with  the  aoent. 
<18:5,6) 

2.  The  initial  documentation  required  for  a 
project  consists  of:  System  Segment  Specification  <SSS) 

<6: — ),  Software  Requirements  Specification  (SRS)  <12: — ), 
Interface  Requirements  Specification  <IRS)  if  numerous 
interfaces  are  required  <13:  — ),  Software  Top  Level  Design 


Document  (STLDD)  <8: — ),  Version  Description  Document  (VDD) 

<9: — ),  Operational  Concept  Document  <OCD>  (11: — ),  and 
Software  Test  Plan  (10: — ).  These  documents  support  basic 
software  engineering  practices.  In  addition  the  documents 
support  major  reviews  and  control  practices  established  in 
MI L-STDs-490  (15: — ),  -483  (14:—),  and  -1521  (16: — ).  The 
documentation  is  initial  insight  into  the  software  that  the 
Government  and  IV  &  V  agent  will  have  to  determine  if 
requirements  are  met. 

3.  The  contractor  shall  conduct  an  initial 
System  Design  Review  (SDR),  as  outlined  by  the  program  office, 
to  initiate  the  product  design  and  development.  Integral  to 
the  SDR  will  be  the  Software  Specification  Review  (SSR).  The 
purpose  is  to  review  essential  requirements,  establish  draft 
baselines,  and  coordinate  interactions  on  deliveries, 
schedules,  costs,  and  other  factors  determined  by  the  program 
office.  (16:23,31)  The  reviews  do  not  replace  the  formal  SDR 
or  SSR  reviews  if  required  in  the  remaining  contract.  The 
initial  review  serves  to  "kick-off"  the  project  effort, 
establishing  common  goals  and  continuity  among  the 
participants.  This  way,  if  changes,  are  made  they  are  made 
against  an  agreed  to  structure,  making  contract  modifications 
easier  and  associated  ramifications  understood. 

This  section  established  an  initial  set  of  tasks  and 
procedures  for  the  Government  and  developer  to  follow; 
providing  a  minimal  level  of  interaction,  coordination,  and 
product  quality  for  a  project.  To  complete  the  common 
guidance  structure,  the  IV  &  V  agent's  tasking  is  required. 

Specific  Guidance  and  Tasks  for  IV  &  V  Aoent.  The  intent 
of  IV  &  V  effort  is  to  detect  problems,  errors,  and 
incompatibilities  with  stated  requirements.  In  addition,  the 
agent  should  evaluate  proposed  products,  related 
documentation,  and  recommended  actions  to  improve  the  product 
to  meet  stated  and  baselined  r equ i r emen ts .  Specific  guidance 
for  the  agent  is  presented  below. 

1.  The  Agent's  RFP  should  contain  a  Not  To  Exceed 
(NTE)  threshold  that  is  finalized  after  the  first  development 
review  or  series  of  reviews,  e.g.,  System  Design  Review  and 
Software  Specification  Review.  This  is  an  alternative  to  the 
BAFO/RFP  release  discussed  earlier.  The  advantage  to  a  NTE  is 
the  agent  can  begin  work  at  the  same  time  as  the  developer,  as 
long  as  the  work  does  net  exceed  a  specified  ceiling.  This 
provides  the  Government  and  agent  time  to  specify  specifically 
what  tasks  are  required.  This  is  performed  after  the  first 
major  program  review,  when  structures  are  established  and 
relationships  are  agreed  to.  Therefore,  effective 
coordination  between  contracts  and  participants  is  maintained. 
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2.  The  Government  needs  to  exercise  caution  in 
selecting  the  IV  6c  V  agent.  The  possibility  exists  -for  legal 
action  i f  the  agent  could  become  a  potential  second  source  tor 
the  software.  This  is  not  always  possible.  Therefore,  early 
agent  establishment  will  surface  probable  conflicts  of 
interest.  The  early  indications  allow  manageable  redirection, 
contract  modifications,  or  reselection  of  an  agent. 

3.  The  agent  must  provide  the  types  of  records  and 
flow  of  communication  required  between  the  Government, 
developer,  and  agent.  Also,  the  extent  of  the  agent's 
participation  in  formal  and  informal  reviews  must  be 
specified.  This  task  is  integrated  with  the  records  task 
since  reviews  usually  require  documentation  and  records. 

(20:5)  The  depth  of  the  documentation  review  and  records 
needs  establishment.  This  avoids  later  confrontations  over 
legal  rights  and  possible  conflict  of  interest  as  noted  under 
the  ASD  Issue  2. 

4.  As  the  development  cycles  mature  into  testing 
phases,  the  validation  tasks  become  critical.  The 
verification  tasks  are  still  required  to  ensure  documentation 
adequately  reflects  the  changes,  improvements,  documented 
results,  trouble  reports,  and  follow-up  actions.  The 
validation  by  the  agent  is  done  in  an  evolutionary  fashion. 
When  the  developer  releases  software  units  the  agent  may  test 
these  lower  units  to  ensure  basic  integrity  of  the  modules. 

As  the  tests  progress  through  units,  to  components,  to  CSCIs, 
and  finally  to  integration  tests,  the  agent  is  in  the  loop. 

The  agent  performs  validation  testing,  examines  test  data 
against  expected  results,  and  traces  requirement  back  to 
governing  specifications  or  documents.  The  verification  and 
validation  is  documented  in  accordance  with  the  established 
records  and  procedures.  <22:57-64) 

5.  The  IV  &  V  contract  should  state  the  requirements 
for  each  phase  or  cycles  in  the  development  process,  based  on 
the  tailoring  specifics  for  the  developer. 

6.  The  agent  should  review,  if  applicable,  the 
Government  OCD  and  CRLCMP.  These  outline  specific  operational 
and  support  issues  for  the  software.  If  the  agent  will  be  the 
supporter  <AFLC) ,  it  is  paramount  that  they  be  involved  in 
operational  concepts.  If  not,  they  may  not  know  support 
issues  or  how  the  system  is  expected  to  operate.  Without 
operational  and  support  knowledge,  the  ability  to  correct  and 
update  software  during  its  lifecycle  is  limited.  (4:55) 

The  above  guidance  provides  an  outline  to  establish  the 
foundation  for  IV  &  V  contracting  and  interactions.  The 
specific  tasks  for  an  IV  &  V  contract  are  outlined  below. 


IV  &  V  Specific  Tasks.  The  tasks  below,  in 
conjunction  with  the  above  guidance,  establish  an  initial  IV  & 
V  effort  in  parallel  with  the  developer.  Tailoring  and 
enhancements  of  various  elements  are  expected,  but  in  concert 
with  the  development  contract. 

1.  The  agent  must  review  developer's  plans, 
spec i f i cal  1 y  the  Software  Development  Plan,  to  ensure  it 
relates  to  the  specific  job  being  performed.  This  is 
essential;  since  as  programs  progress,  design  and 
development  must  track  the  plan. 

2.  During  design  the  agent  is  tasked  to  review 
the  applicable  SSS  (or  equivalent),  OCD  and  SRS/IRS  documents 
to  ensure  proper  flow  down  of  requirements.  As  an  example, 
the  agent  ensures  requirements  in  the  STLDD  flow  to  SRS  which 
flow  to  the  SSS.  <8:4,4;  12:9,14;  4:8,18) 

3.  In  conjunction  with  specification  review  the 
agent  should  review  interfaces,  ensure  their  proper  flow  in 
requ i r emen ts ,  and  review  new  interfaces  and  CSCIs  as  they 
develop.  This  ensures  new  requirements  are  not  i nadver ten t 1 y 
introduced  by  changes.  This  is  known  as  requirements  creep  in 
the  acquisition  field  and  is  watched  closely  to  ensure 
software  at  the  end  meets  the  established  requ i remen ts . 

4.  The  IV  6c  V  agent  must  describe 
organizational  structures  and  relationships  for  the  agent's 
team.  This  includes  basic  agreements  on  exchange  of  data  with 
the  software  developer.  Additionally,  an  expected  list  of 
resources  required  to  perform  the  IV  &  V  tasks  is  developed. 
(4:81)  This  factor  includes  the  personnel,  facilities, 
environments,  Government  related  data,  facilities,  software, 
and  equipment  required  or  expected.  (20:4)  The  agent  delivers 
the  information  in  a  Software  IV  Ik  V  Plan,  similar  to  the  SDP 
structure,  but  in  contractor  format.  This  is  an  option  until 

a  formal  IV  &  V  plan  for  software  is  approved. 

5.  The  agent  requires  a  copy  of  the  VDD  to 
ensure  versions  being  developed  and  matured  by  the  developer 
are  properly  utilized  in  the  noted  fashions.  Additionally, 
the  agent  needs  to  develop  specific  documentation,  in  the  form 
of  an  SDP  and  test  plans,  documenting  how  the  agent  will  do 
software  development  and  testing  as  applicable. 

4.  The  agent  must  prepare  to  attend  the  first 
two  project  reviews,  e.g.,  SDR,  SSR.  Before  attending  the 
reviews,  the  agent  presents  inputs  about  the  developer's 
project  to  the  Government  for  consideration.  The  inputs  are 
based  on  documents  reviewed  and  requirements  outlined  in  MIL- 
STD-1521.  (14:23,31) 


5 

H 

i 


rjtvu*. 


*  rf  n rjw  nr/ww\r^nnnrw  yr»  vu  ir. 


Summary 

The  Category  I  solution  provides  the  product  divisions  the 
means  to  improve  IV  &  V  implementations.  The  solution 
addresses  three  areas;  General,  Developer,  and  IV  &  V 
Guidance.  These  areas  are  applicable  to  the  Government, 
software  developer,  and  IV  &  V  agent.  Without  a  coordinated 
effort  among  the  participants,  the  IV  &  V  effort  becomes  a 
cost  and  schedule  hindrance  or  a  useless  tool.  With  a 
coordinated  approach,  IV  &  V  supports  the  software  development 
ef f or  t . 


CATEGORY  II 

DATA  COLLECTION  AND  ADVOCACY 

Category  II  concerns  software  issues  involving  collecting 
data  and  maintaining  advocacy  for  software  IV  &  V  across  Air 
Force  Systems  Command.  The  issues  that  produced  this  category 
were  derived  from  four  sources;  ESD,  SD,  AD,  and  AFLC;  as 
summarized  in  Table  1.  The  following  presents  each  issue,  its 
relationship  to  the  category  and  a  proposed  solution  for  the 
Category  II  concerns. 

Product  Division  Issues 

The  issues  derived  from  the  product  divisions  are  not 
restricted  to  the  divisions  themselves.  Their  concerns,  in 
the  authors  view,  echo  a  basic  theme  across  AFSC.  Without 
backing  from  headquarters,  the  field  can  not  always  to  what  is 
right,  especially  in  terms  of  cost  and  schedules.  The  field 
does  not  need  additional  oversight,  but  an  avenue  to  share 
ideas  and  experiences.  This  allows  direction  and  policy 
formulation  to  evolve  outside  of  a  vacuum.  Therefore,  the 
following  information  presents  the  issues  which  establish 
Category  1 1 . 

ESD  Issue  6.  The  ESD  issue  suggests  a  survey  be  employed 
to  monitor  the  effectiveness  of  their  implementation  of  an 
"on-call"  IV  &  V  agent.  The  survey  would  provide  data  for 
judging  the  positive  and  negative  aspects  of  the  agents  work. 
Since  the  "on-call"  agent  is  a  new  approach,  the  effort's  true 
value  is  unknown.  The  value  could  effect  ESD  efforts  only  or 
be  applicable  for  other  AFSC  product  divisions.  Without  data, 
the  effectiveness  of  the  program  and  possible  corrections  or 
improvements  applicable  for  other  divisions  is  unknown. 

AFCMD  I  ssue  6.1.  Air  Force  Contract  Management  Division's 
concern  parallels  that  of  ESD's  Issue  6.  A  survey  would 
provide  data  on  current  IV  &  V  implementations.  The  data 
supports  identification  of  potential  problems  and  solutions. 
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Also,  the  survey  supports  AFCMD  Issue  8,  concerning  the 
establishment  of  contractual  responsibilities  and  avoiding 
duplication  of  efforts.  The  survey  provides  visibility  of 
program's  early  objectives  and  i n ter-re 1  at i onsh i ps  among  major 
participants. 

AFLC  Issue  6.2.  Air  Force  Logistics  Command  is  developing 
a  survey  to  address  AFLC  IV  fit  V  issues  for  software.  However, 
the  usefulness  of  the  data  is  applicable  to  AFSC  programs  and 
supports  AFSC  Issues  6  and  6.1.  The  AFLC  office  heading  this 
effort  provides  additional  assistance  to  the  computer  resource 
office  at  HQ  AFSC  under  the  Joint  Logistics  Commanders'  working 
groups.  <33: — )  The  exchange  of  information  from  surveys 
would  provide  valuable  data  to  both  commands,  especially  since 
AFLC  is  not  normal  1 y  involved  in  early  phases  of  AFSC 
projects.  After  AFSC  turns  the  projects  over  to  AFLC,  AFSC  is 
removed  from  day  to  day  project  support  and  field  corrections. 
Therefore,  an  exchange  of  data  would  provide  both  commands  the 
opportunity  to  review  IV  fit  V  data  and  evaluate  effectiveness, 
policy  options,  and  common  IV  &  V  practices. 

SD  Issue  7.  Space  Division  expressed  concern  that 
software  developers  were  using  the  IV  &  V  agent  as  their  own 
quality  evaluation  function.  This  relieved  the  developer  from 
doing  the  required  evaluations,  which  cost  time  and  money. 

Thus,  the  product  was  delivered  on  time,  but  often  with  errors 
which  required  more  time  and  money  to  fix.  The  SD  issue  seeks 
to  improve  the  software  management  interface.  This  issue  was 
discussed  under  Category  I  for  ways  to  improve  the  interface. 
However,  HQ  AFSC  awareness  of  these  potential  conflicts  is 
paramount,  since  budgets,  schedules,  and  performance  are  their 
prime  concern.  Therefore,  a  survey  collecting  and  a  central 
office  monitoring  relevant  IV  &  V  data  could  support  the 
awareness.  This  provides  initial  visibility  to  ensure 
policies  are  effective  and  guidance  supports  contractual 
requirements  under  Category  I. 

AD  I ssue  1 0 .  Armament  Division's  concern  is  that,  without 
a  strong  advocate  for  IV  &  V  early  in  a  programs  1 ife,  IV  fit  V 
for  software  would  be  overlooked.  When  problems  do  surface 
with  the  software,  during  preliminary  and/or  operational 
testing,  IV  fit  V  type  tasking  is  initiated  to  find  the  errors. 

At  this  point  software  and  hardware  are  integrated  to  a  high 
degree  and  any  changes  may  adversely  effect  the  performance, 
causing  additional  schedule  and  cost  impacts.  (4:3?)  Software 
IV  fit  V  provides  an  avenue  to  catch  errors  early  before  they 
become  integrated  within  the  hardware  and  system  level 
products.  The  ability  for  early  detection  is  achievable  if 
guidance  on  IV  fit  V  implementation  (discussed  under  Category  I) 
and  an  advocate  that  supports  the  early  implementation  is 
available.  Headquarters  AFSC  is  ideally  suited  for  this 
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advocate  position.  Headquarters  provides  initial  direction 
and  requirements  to  product  divisions  and  associated  program 
offices  concerning  projects.  In  addition,  the  computer 
resource  offices  at  each  product  division  form  a  MCCR  network 
that  HQ  AFSC  interacts  with.  This  network  provides  the  first 
step  in  maintaining  the  flow  of  information,  direction,  and 
advocacy  for  programs. 

Category  II  Requirement 

The  above  issues  establish  the  need  for  feedback  and 
advocacy  on  IV  &  V  for  software.  A  survey  supports  the 
feedback  process.  Using  established  networks,  the  survey  can 
proceed  to  collect  IV  &  V  data  without  additional  manpower. 
The  pinnacle  of  the  network  must  advocate  early  use  of  IV  &  V 
to  avoid  potential  problems  of  catching  errors  late.  Late 
errors  require  extensive  corrections  and  adversely  effects 
software  and/or  system  design  to  some  degree.  (4:3?> 


Category  II  Proposed  Solution  -  Four  Interacting  Tools 

There  are  four  tools  that  could  support  the  issues 
outlined  above.  The  tools  cover  specific  offices  of  primary 
responsibility  <OPR),  using  at  their  disposal,  already 
established  practices  and  relationships.  Also,  collecting 
data  serves  as  a  tool  using  established  relationships.  Each 
of  these  tools  are  presented  below  with  accompanying  rational. 

Each  tool  interacts  with  the  others.  They  form  a 
collective  group  of  processes  that  help  in  establishing  IV  &  V 
techniques  across  AFSC.  The  techniques  evaluate  effectiveness 
and  seek  to  improve  IV  &  V  concerns  before  they  effect  the 
reliability  of  a  desired  system. 

Tool  1  -  Interaction.  The  ability  to  interact  between 
headquarters  and  product  divisions  is  paramount.  With 
interaction,  the  ability  to  evaluate  problems,  share  lessons 
learned,  and  track  applications  is  available  to  all  parties. 
Currently,  HQ  AFSC/PLR,  Mission  Critical  Computer  Resources 
Directorate,  is  responsible  in  establishing  the  MCCR  policy 
for  AFSC,  within  AFR  800-14.  This  office  coordinates  with 
individuals  at  each  product  division  called  the  Computer 
Resource  Focal  Point  <CRFP>.  It  is  the  CRFP's  responsibility 
to  work  with  the  respective  program  offices  and  provide  MCCR 
advice  on  projects.  The  CRFP's  advice  is  directed  through  the 
Computer  Resource  Working  Groups  (CRWG)  and  development  of 
Computer  Resource  Life  Cycle  Management  Plans  (CRLCMP)  for 
programs.  The  CRWG  and  CRLCMP  are  tasked  directly  out  of  AFR 
800-14.  HQ  AFSC/PLR  monitors  the  CRFPs  through  bi-annual 
meetings.  Therefore,  the  application  of  software  policy  and 
the  effects  of  the  IV  &  V  core  requirements  is  assessable. 
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Tool  2  -  Early  Application.  Early  application  of  IV  &  V 
starts  at  the  headquarters.  When  a  program  is  initiated,  so  is 
the  paper  work  to  establish  the  program  baseline,  budgets,  and 
schedules.  Numerous  offices  at  HQ  AFSC  review  the  program 
documentation  and  recommend  program  actions  on  a  AFSC  Form  56. 
HQ  AFSC/PLR  provides  MCCR  direction  via  this  form.  Areas  of 
interest  include  typical  MCCR  documents  to  be  obtained  (SRS, 
SDP,  CRLCMP,  etc.)  and  interactions  to  be  performed  (CRWG). 

The  PLR  office  has  a  unique  opportunity  to  promote  IV  &  V 
actions  for  further  cons i derat i on  through  this  program 
documentation.  Being  responsible  for  MCCR  policy  and  CRFP 
interactions,  the  PLR  office  is  prime  for  advocating  IV  &  V. 
This  way,  the  product  division  CRFPs  have  an  avenue  to  employ 
the  IV  Sc  V  concepts  that  have  support  from  headquarters. 

Using  direction  from  headquarters,  the  CRFPs  and  program 
offices  can  evaluated  the  common  guidance  for  and  scope  of  IV 
&  V  efforts. 

Tool  3  -  Data  Collection.  The  software  environment  is 
dynamic.  New  techniques  < DQD-STD-21 67) ,  languages  (Ada),  and 
concepts  (artificial  intelligence)  are  advancing  the 
technology  that  programs  employ  as  they  go  into  the  Air  Force 
inventory.  The  ability  to  react  to  new  innovations  is 
critical.  The  time  required  to  incorporate  new  approaches  is 
often  longer  than  the  innovations  life.  To  decrease  the  time 
interval  requires  the  Air  Force  to  know  when  and  where  to 
improve  each  system.  To  do  this,  in  part,  requires  sound 
engineering  practices  and  evaluations.  An  effective  IV  &  V 
approach  provides  a  measure,  not  100  percent,  that  a  program 
can  perform  as  required.  This  measure  of  success  is  the  first 
step  in  integrating  new  approaches.  Without  key  understanding 
of  the  parts  of  a  problem  and  ability  to  react  to  change,  new 
innovations  are  not  integrated  in  a  timely  manner.  Therefore, 
current  data  on  the  IV  &  V  implementation  across  AFSC  is 
required  to  support  improved  practices. 
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HQ  AFSC/PLR  is  the  prime  candidate  for  OPR  to  collect  the 
data  on  IV  &  V  implementation.  This  office  supports  and 
drafts  policy  changes.  It  also  coordinates  with  division 
counter  parts  and  evaluates  technology  improvements  in  current 
and  future  systems.  The  structure  is  in  place  to  collect  the 
data  via  the  CRFPs,  bi-annual  meetings,  and  reviews  of  program 
and  associated  documentation.  Appendix  A  represents  a  survey, 
developed  by  the  author,  which  provides  the  data  collection 
function,  allowing  the  OPR  to  establish  cognizance  over  the  IV 
&  V  implementations.  The  survey  serves  a  secondary  function 
of  monitoring  improvements  to  software  development  and  quality 
areas.  The  entire  software  process  encompasses  development, 
quality,  and  IV  &  V.  As  improvements  to  these  areas  arise, 
the  survey  becomes  a  management  tool  to  better  evaluate  the 
implications  of  new  technology.  It  should  not  become  a 
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bureaucratic  paper  exercise,  but  a  critical  data  point  on  the 
application  and  implementation  of  key  processes.  With  HQ 
AFSC/PLR  directing  the  survey,  a  common  -focal  point  is 
available  to  evaluate  data,  present  results,  and  develop 
recommendations.  This  is  done  with  the  aid  o-f  -field  inputs 
and  -field  review. 

Tool  4  -  Fol  1  ow  Up  .  An  e-f-fective  process  has  outside 
-feedback  -for  evaluation.  To  adequately  assess  the  Category  I 
and  II  applications,  a  process  o-f  evaluation  is  required.  The 
HQ  AFSC  Inspector  General  <IG>  serves  this  unique  -function. 

The  IG  assesses  program  and  other  related  o-f-fice  functions 
across  AFSC.  The  IG's  purpose  for  IV  &  V  should  be  to  assess 
the  impact  of  IV  it  V  applications.  These  applications  cover 
the  utility  of  having  an  IV  6c  V  agent,  use  of  common 
requ i remen ts ,  and  pay  back  in  producing  an  effective  software 
product  within  the  cost  and  schedule  constraints.  The  IG's 
analysis  is  an  additional  data  point  allowing  PLR  to  assess 
the  usefulness  of  the  IV  &  V  efforts. 

Summar y 

The  Category  II  solution  addressed  the  needs  of  the 
product  divisions  to  monitor  and  improve  the  IV  &  V  structure. 
The  structure  involved  maintaining  sufficient  control  over  the 
IV  &  V  process.  Without  up-to-date  information  on  IV  &  V,  the 
dynamic  nature  of  software  development  and  technologies  could 
adversely  effect  the  software  intensive  systems.  Therefore, 
an  advocate  of  the  process  provides  guidance  and  direction  to 
improve  IV  &  V  implementations.  Using  a  survey  to  collect  the 
data,  technology  insertion  assessments,  CRFP/program  office 
inputs,  and  AFSC/IG  assessment  produces  a  closed  loop  solution 
for  Category  II.  The  unique  item  about  the  solution  is  it 
comes  with  a  structure  already  in  place.  The  OPR  is  in 
existence,  MCCR  network  of  CRFPs  is  working,  and  the  IG  is 
reviewing  programs  for  compliance  with  established  MCCR 
guidance.  The  only  added  feature  is  the  survey,  outlined  in 
Appendix  A. 


INTERIM  CONCLUSION 

This  process  of  common  guidance,  OPR/CRFPs,  surveys,  and 
follow-up  evaluations  tie  the  software  development  process 
into  a  workable  structure.  Categories  I  ar II  provide  common 
guidance,  data  collection,  and  advocacy  aveiues.  To 
effectively  support  the  software  IV  &  V,  each  category 
addressed  key  issues  and  players.  In  any  contractual 
development,  the  Government,  developer,  and  IV  &  V  agent  have 
critical  functions  for  effective  software  development.  In 
addition,  to  track  developments  across  AFSC  requires  a  network 
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and  -feedback  to  ensure  policies  and  guidance  are  su-f -f  i  c  i  en  t . 
The  Category  I  and  II  solutions  cover  these  areas.  It  will 
cost  time  in  extra  man  hours  and  project  dollars  to  implement. 
But  the  savings  in  ability  to  react  faster  and  provide  usable 
systems  having  lower  error  rates  is  worth  the  cost.  It  is 
estimated  that  by  using  IV  4  y  a  5'/.  to  25'/.  cost  avoidance  is 
achievable  on  development  contracts.  <4:100)  The  average  cost 
of  an  intensive  IV  6c  V  effort  is  figured  at  25X  of  the 
respective  development  effort.  <4:100)  Therefore,  IV  Sc  V 
efforts  in  a  variety  of  products  can  pay  their  own  way. 
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Chapter  Five 


SUMMARY 

CONCLUSIONS 

The  IV  &  V  implementations  across  AFSC  are  varied,  in 
part,  to  match  various  products  ranging  -from  aircraft, 
missiles,  and  electronics,  to  space  systems.  In  addition,  the 
variance  reflects  respective  experiences.  Each  IV  &  V 
implementation  reviewed  was  in-line  with  the  intent  of  current 
direction.  New  policy  or  regulations  are  not  required  to 
improve  IV  &  V  implementations.  By  applying  a  single  software 
development  standard  < DOD-STD-21 67) ,  software  IV  &  V  can 
approach  a  standard  implementation.  This  implementation 
should  exhibit  less  variances  when  a  common  IV  &  V  structure 
is  integrated  with  current  development  practices.  However,  to 
ensure  a  common  structure,  in  conjunction  with  DOD-STD-21 67 , 
feedback  and  control  of  IV  &  V  implementations  are  required. 

The  structure  developed  in  Chapter  Four  is  the  first  step 
in  developing  a  standard  IV  &  V  implementation  based  on  DOD- 
STD-2167.  The  structure  consists  of  common  IV  &  V  guidance 
and  four  tools  for  improving  IV  &  V  implementations  across  Air 
Force  Systems  Command.  The  guidance  establishes  initial  IV  & 

V  requirements  for  a  program  office.  A  feedback  loop  is 
available  through  one  of  four  tools;  from  monitoring  IV  &  V 
implementations  to  improving  guidance  through  associated  OPRs . 
This  ensures  up-to-date  inputs  are  available  for  the  program 
offices.  Additionally,  IV  &  V  data  from  the  program  offices 
are  available  to  HQ  AFSC. 

A  common  structure  permits  AFSC  to  evaluate  various  IV  &  V 
implementations  using  a  common  set  of  criteria.  The 
evaluation  is  through  collection  of  relevant  data  using  a 
network  of  interested  participants.  The  four  tools  provide 
the  appropriate  avenues  for  collection  of  the  data  using  the 
CRFP  network,  headquarters  directorate  to  correlate  and 
disseminate  the  data,  and  the  AFSC/IG  to  review  the  IV  &  V 
impacts.  The  IV  &  V  evaluations  provide  the  ability  to 
improve  the  respective  policy  and  implementation  activities. 

IV  &  V  implementation  yields  positive  results  on  programs 
if  appl  ied  early.  < 4 : 4 , 3? ; 1 : 2-5)  To  that  end,  AFR  800-14 
directs  the  use  as  early  as  possible.  ( 4  : 4 , 3? ; 1 : 2-5)  The 


Joint  Logistics  Commanders  have  initiated  tasks  to  develop  a 
guidebook  and  standards  that  address  IV  6c  V  implementation. 
However,  that  direction  is  not  yet  complete.  Therefore,  the 
objective  of  establishing  IV  6c  V  early  in  a  program  is  met 
through  the  application  of  Category  I  and  II  solutions. 


IMPLEMENTATION 


To  apply  the  solutions  mentioned  above  the  following 
implementation  is  recommended. 

1.  HQ  AFSC/PLR  should  coordinate  with  the  CRFPs  on 
the  common  IV  6c  V  structure.  They  should  review  the  basic 
contents  to  ensure  applicability  to  respective  product 
division  applications. 

2.  HQ  AFSC/PLR  should  send  the  survey,  Appendix  A, 
to  each  CRFP  to  establish  the  IV  6c  V  information  base.  In 
addition,  AFSC  should  coordinate  the  data  base  with  associated 
AFLC  survey  data.  Then  review  both  information  surveys  during 
the  bi-annual  CRFP  and  HQ  AFSC/PLR  meeting.  The  result  of  the 
review  is  used  to  improve  IV  6c  V  direction.  At  the  next  bi¬ 
annual  meeting  IV  &  V  improvements  are  evaluated.  This 
approach  provides  a  means  to  examine  issues  with  interested 
participants.  Then  the  respective  focal  points  and 
headquarters  can  develop  solutions  in  conjunction  with  their 
program  offices.  The  real-time  effect  of  working  solutions 
with  current  programs  provides  real-world  solutions  to  policy 
development,  if  required. 

3.  HQ  AFSC/PLR  should  coordinate  with  HQ  AFSC/CC  on 
how  to  manage  IV  6c  V  implementations.  This  can  be 
accomplished  via  an  AFSC  pamphlet.  The  pamphlet  should  cover 
specific  tasks  and  processes  for  IV  6c  V.  In  addition,  it 
should  discuss  how  to  scope  IV  &  V  efforts  to  match  particular 
projects.  This  paper  can  provide  the  essentials  for  these  two 
areas . 


4.  HQ  AFSC/IG  should  initiate  a  software  IV  &  V 
special  interest  item  for  their  program  reviews.  This 
provides  the  IG  field  unit  with  direction  on  how  to  evaluate 
software  IV  6c  V  on  programs. 

5.  HQ  AFSC/PLR,  through  representation  on  the  JLC- 
JPCG-CRM,  should  support  completion  of  the  guidebook  on 
software  IV  &  V.  A  companion  item  is  to  support  and 
coordinate  publication  of  the  software  development  and  quality 
standards  < DOD-STD-21 67A  and  DOD-STD-21 68) .  This  avenue 
allows  integration  of  key  elements  of  the  software  IV  6c  V 
guidance,  Categories  I  and  II. 
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EXPECTED  RESULTS 

By  providing  common  IV  &  V  guidance,  -for  the  variety  of 
AFSC  products,  a  common  -foundation  is  established.  Since 
products  di-f-fer  the  -foundation  can  grow  to  -fit  the  individual 
requirements.  The  availability  o-f  a  feedback  network  allows 
guidance  to  support  what  the  field  needs.  This  provides 
balance  between  the  common  guidance,  policy,  budget,  schedule, 
and  performance  contraints. 
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APPENDIX  A 


INFORMATION  SURVEY 
ON 

SOFTWARE  INDEPENDENT  VERIFICATION  AND  VALIDATION 


The  following  survey  is  intended  to  collect  data  on  the 
usefulness  of  software  Independent  Verification  and  Validation 
(IV  4c  V)  on  major  programs  within  Air  Force  Systems  Command. 
The  data  will  be  used  to  review  IV  &  V  direction,  problems, 
improvements,  and  importance.  An  analysis  will  determine  if 
policy  direction  is  adequate  or  additional  guidance  should  be 
considered  to  ensure  the  proper  use  of  IV  4c  V. 


1.  Program  using  IV  4c  V; 

2.  Lifecycle  phase  of  the  program: 

3.  CRLCMP  status  and  IV  4c  V  recommendations: 


4.  Agencies  and  contractors  involved:  (SPO,  AFPRO,  developer, 
IV  4c  V  agent,  0T4cE,  AFLC,  and  user) 


5.  Fiscal  year  award  of  developer  and  IV  4c  V  con trac t < s)/MOA 
as  applicable:  (Type  of  contract  awarded  for  each;  Fixed 
Price,  Cost  Plus,  etc.) 


6.  Cost  of  the  IV  4c  V  or  percentage  of  program  cost  devoted 
to  IV  4c  V  per  fiscal  year: 


7.  List  applicable  software  standards,  metrics,  and  direction 
implemented  on  the  program  for  both  the  developer  and  IV 
4c  V  agent  as  applicable: 
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8.  IV  4c  V  agent  qual  i  -f  i  cat  i  ons  : 


a.  Experience  level  of  IV  4<  V  personnel. 


b.  Number  of  personnel  involved  (full  and  part-time) 


IV  4c  V  agent  interaction  with  the  prime  developer: 


a.  Level  of  access  to  developer's  software,  associated 
tools,  and  environments. 


b.  Directly  submitted  write-ups. 


c.  Inputs  sent  to  the  program  office 


d.  Subcontracted  to  the  prime 


10.  List  the  available  guidance  which  was  followed  in 

determining  IV  4c  V  use:  (Headquarters  direction,  user, 
support  agent,  Computer  Resource  Working  Group,  etc.) 


a.  Specifics  that  determined  IV  4c  V  scope. 


b.  Products  required  to  be  submitted  by  the  IV  4c  V  agent 
to  the  contracting  agency  (Government  or  prime 
deve 1 oper ) 


c.  Software  indicators  (management  and  quality)  used  by 
the  program  office. 


11.  Critical  level  of  the  software  and  system:  (safety  of 
flight,  weapon  reliability,  etc.) 


12.  Software  complexity: 


a.  Interrelationships  with  other  computer  software  in 
the  mission-critical  system  (air  or  ground  base). 


b.  Number  of  interrelated  Computer  Software 

Configuration  I  terns  within  the  mission-critical 
system . 


c.  Interface  control  requirements  for  the  critical 
system  (aircraft,  missile,  avionics,  ground  base, 
etc.). 


d.  Software  language(s)  employed, 
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Software  Verification  Actions: 

a.  Number  of  lines  of  code  inspected  (or  to  be). 

b.  Testing  level  of  CSCs  and  CSCIs  (unit,  integration, 
etc)  . 

c.  Number  and  types  of  documents  to  be  reviewed. 

d.  Number  of  error s/concer ns  reported. 

e.  Number  of  reported  error s/concerns  fixed  and  delayed. 


14.  Software  Validation  Actions:  (if  applicable) 

a.  Testing  agency  or  contractor. 

b.  Types  of  tests  performed. 

c.  Level  of  testing  to  the  system  requirements. 

d.  Acceptability  of  test. 

e.  Additional  documents  and  code  reviews  performed. 

f.  Number  of  errors  or  corrections  detected. 

g.  Number  of  errors/corrections  fixed  and  delayed. 


15.  What  affect  did  or  will  the  IV  &  V  agent  have  on  the 
overall  software  development  effort: 


1<4.  The  interaction  with  the  respective  CAS  activity.  List 
the  functions  from  above  that  the  CAS  activity  did  or 
could  provide  or  manage  for  the  Government: 


17.  Additional  areas  of  interest  concerning  use  or 
improvements  for  IV  &  V  implementation: 


